remotemanager.dataset.dependency module

The Dependency module handles the chaining of Datasets.

It provides overrides for any linked Dataset to propagate required calls.

class remotemanager.dataset.dependency.Dependency

Main Dependency class.

Note

This class is internal and should not be interacted with directly.

Handles the chaining of Datasets and stores the “network”. It provides overrides for any linked Dataset to propagate required calls.

add_edge(primary: Dataset, secondary: Dataset) → None

Adds an “edge” to the dependency network.

This edge represents a parent-child link.

Parameters:

• primary (Dataset) – Parent Dataset object.

• secondary (Dataset) – Child Dataset object.

property network: List[Tuple[Dataset, Dataset]]

Returns the internal dependency network.

This is a list of tuples representing the parent-child relationship.

Returns:

List(Tuple(Dataset, Dataset)

get_children(dataset: Dataset) → List[Dataset]

Collect and return all children for a given Dataset object.

Parameters:

dataset (Dataset) – Dataset to query.

Returns:

List of children of the given Dataset.

Return type:

List

get_parents(dataset: Dataset) → List[Dataset]

Collect and return all parents for a given Dataset object.

Parameters:

dataset (Dataset) – Dataset to query.

Returns:

List of parents of the given Dataset.

Return type:

List

property ds_list: List[Dataset]

Returns a flattened list of all datasets within the network.

Returns:

_description_

Return type:

_type_

property runners: List

Returns a list of all runners within the network.

update_db()

Ensure all datasets within a dependency update their databases together

remove_run(ident: int | str | dict) → bool

Chains the Dataset.remove_run function to all Datasets within the dependency.

Parameters:

ident (Union[int, str, dict]) – Runner identifier.

Returns:

True if all runners were removed successfully, False otherwise.

Return type:

bool

clear_runs() → None

Remove all Runners from the Datasets.

clear_results(wipe: bool) → None

Ensure all datasets within a dependency clear their results together

Parameters:

wipe (bool) – Additionally delete the local resultfiles if True

wipe_local(files_only: bool = False) → None

Clears out the local directory.

Parameters:

• files_only (bool, optional) – Attepts to only delete files.

• False (Defaults to)

• directory. (deleting the)

wipe_remote(files_only: bool = False) → None

Clears out the remote directory.

Parameters:

• files_only (bool, optional) – Attepts to only delete files.

• False (Defaults to)

• directory. (deleting the)

hard_reset(files_only: bool = False) → None

Hard resets all datasets. This will wipe local and remote directories.

Parameters:

• files_only (bool, optional) – Attepts to only delete files.

• False (Defaults to)

• directory. (deleting the)

append_run(caller, chain_run_args, run_args, force, lazy, *args, **kwargs)

Appends runs with the same args to all datasets

Parameters:

• lazy (bool)

• caller – (Dataset): The dataset which defers to the dependency

• chain_run_args (bool) – for dependency runs, will not propagate run_args to other datasets in the chain if False (defaults True)

• run_args (dict) – runner arguments

• force (bool) – force append if True

• lazy – do not update the database after this append (ensure you call update_db() after appends are complete, or use the lazy_append() contex)

• *args – append_run args

• **kwargs – append_run keyword args

Returns:

None

finish_append() → None

Propagates the completion of runner append to all datasets in the chain.

static get_runner_remote_filepath(runner, workdir: str, filetype: str) → str

Generates the relative remote path from workdir to a runner file The file is specified by filetype

stage(force: bool = False, extra: str = '', force_ignores_success: bool = False, verbose: Verbosity | None = None, **run_args) → bool

Handles the Dataset Staging for a dependency situation. :param dry_run: create files, but do not run :type dry_run: bool :param extra: str

extra text to add to runner jobscripts

Parameters:

• force_ignores_success (bool) – If True, force will submit only failed runners

• verbose – Sets local verbose level

transfer(force: bool = False, extra: str = '', force_ignores_success: bool = False, verbose: Verbosity | None = None, **run_args) → bool

Transfer the files to the remote

run(force: bool = False, dry_run: bool = False, verbose: None | int | bool | Verbosity = None, extra: str = '', force_ignores_success: bool = False, **run_args) → bool

Handles the Dataset Run for a dependency situation.

Parameters:

• dry_run (bool) – create files, but do not run

• extra – str extra text to add to runner jobscripts

• force_ignores_success (bool) – If True, force will submit only failed runners

• verbose – Sets local verbose level

update_runners()

Manifest only needs to be collected once, then all the runners can be updated by that call

check_failure()

Raises a RuntimeError if an error is detected in any of the runners

Relies on the runner.is_failed property